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“Good judgment comes from experience, 
and experience comes from bad judgment.” 
— Frederick P. Brooks © 2020 Philip Koopman 1 
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Lifecycle Issues is aes 


= Anti-Patterns: mewn) eo 
e No version control 7a 
e No configuration management 
e Incremental features without baseline 











DEVELOPMENT 


PRODUCT 


MANUFACTURING 
DESIGN PROCESS 


UPGRADES 
DESIGN 


SUPPORT/ 
MAINTENANCE 


= Lifecycle issues DEPLOWENT 
e Version control: keeping different versions straight 
e Configuration management: what's in the deployed system 
e Lifecycle: old embedded systems (almost) never die 
— Spare parts for obsolete systems 
— Mid-life upgrades 


PRODUCTION 
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INSTALLATION ERROR — 


Airbus confirms software configuration 
error caused plane crash 


Airbus A400M flight recorder data confirms "quality issue" in setup caused failure. 


SEAN GALLAGHER - 6/1/2015, 12:38 PM 
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https://www. youtube.com/watch?v=TUiX6m6WLdY &ab_channel=Sciences 


The Register 6/10/75: 

“Torque calibration parameters were wiped ... 
“Pilots only get warning above 120 meters off 
SSE a eae eS ES the ground” 


As Ars reported on May 19, Airbus had issued a warning to its military customers about a 
potential software problem in the engine control software for the A400M. The release of the 
exact cause of the crash, however, had been delayed because a Spanish magistrate placed the 





: ' : ; ' : http://arstechnica.com/information- 
flignt data recorders from the aircraft under seal. Airbus has since been able to obtain the flight technology’ 2015/06 airbus-Confirme-eoftware- 
data, which Lahoud said confirms that the engine control software had been improperly configuration-error-caused-plane-crash/ 


configured during the installation of the engines on the ill-fated aircraft. 
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Version Control el 
m Stores & navigates snapshots of versions a sever 1 fork 
e Ability to roll-back to previous version a OS 
e Synchronizes compatible versions , . ho ™ {=| 
— Development vs. release versions \ 





— “Branch” and “merge’ parallel efforts 
e Given a version number, give me the software 





6 git checkou&mast 
*git pull upstream master 


Ny 


4. git push -u origin feature/new-service 


m Many popular tools — use one! 
e Watch out for binary file management 
e Multi-site can require special care 










— 3. git checkout -b feature/new-service master 
7 7 it checkout feature/mega-feature 
t rebase master 


Paes 


ypenclipart.org/detail/278845/git-opensource-workflow 


m Beyond the obvious... 
e Also version: design documents, requirements, tests, tool chain 
e Need process for who/what/when to update version 
e Needs to tie into disaster recovery (have you tested it?) © 2020 Philip Koopman 4 


=m CM is identifying a particular version 


m Beyond the obvious... 
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Which version has which bug fixes? 
Which version has which features? 

Which version should be the next release? 
Which version is in a failed product? 





Snapshot every build and record its manifest | 

Make sure returned units can self-report version Airbus A-380 bolt with part tracking 
information. (Size: 2 cm x 1 cm) 

— Which library version, which driver version, etc. 

— Need SW version, HW version, config data version (if applicable) 

— Attempts to track this in a central database never work 100% 

Need to know which SW is compatible with which HW 

— Need to be able to re-create a build, which might require obsolete tools 

Feature activation: which features have been licensed by user? =—- ©2020 Philip Koopman 5 


Gas Pump Startup Printout Example 
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CLAMSHELL PRINTER POARD 
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xk CONFIGURATION «x 


~- Revisio VO0507 
- CHKS/CRC : (C2B1) 5517h 
~ S/N : y¥/W*-SSSS 
- Pre Heating : on 
- Paper Temp. . high 
- Auto Advance - on 
~ Watchdog : on 
- Opto Jam s/h: CC /99 
-~ Opto Pass s/h: 28/084 


*x* SERIAL INTERFACE 


- Parameters . n,8,1 
- Flow Control : Dtr/Dsr 
- Baud Rate : 38400 
- Level 1/0 : TTL 

L" #S%E ” () es = /01.°34567 
89::<= 5 2@ABCDEFGH I JRLMNO 


PORSTUVWXYZE\ 17” abcdefg 
hijJklmnopaqrstuvwxyse {if - 


Carnegie 
Mellon 
University 


(Aug 2009) © 2020 Philip Koopman 6 


Carnegie 






Embedded Systems Live (almost) Forever ae 
m= 10-50 year lifecycles are common al at —— eee 


e Example: SAGE air defense system 

— Started 1954; deployed 1963 

— Vacuum tubes; 500,000 lines of code 

— In operation until 1983 
e 1AESS telephone switch: 1976 through 2008+ 
e Aircraft can operate for 25-30 years 

Cars routinely operate for 15+ years 


=m Challenges 
e Disaster recovery (includes vendor bankruptcy) 
e End-of-life for hardware & software 
e Adding new services to existing platform 
e Security problems if not updated © 2020 Philip cchomene? 








Britain’s Doomsday Nuke Subs 
Still Run Windows XP 


The fate of the country’s nukes is in the hands of an obsolete operating system. 


‘a By Kyle Mizokami Jan 21, 2016 


Pp 





https://www.popularmechanics.com/military/weapons/a19061/britain 
s-doomsday-subs-run-windows-xp/ © 2020 Philip Koopman 8 
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Posted on April 26, 2018 at 12:59 PM 


WINDOWS 95 POWERED MEDICAL 
EQUIPMENT ARE BEING HIT BY HACKERS 


https://koddos.net/blog/windows-95-powered-medical- 
equipment-are-being-hit-by-hackers/ © 2020 Philip Koopman 9 
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m Take CM & Versioning seriously — they are different 
e Use a formal Build Process to release 
— Check that all versions in build are correct; record manifest 
e Plan for 2x the lifecycle you think will happen 
— And have a plan for product end of life 





m Lifecycle pitfalls 
e Releasing the wrong version 
—- Development version, debug version (e.g., watchdog turned off), etc. 
e Cutting corners on software configuration management 
— What if you can't reconstruct software involved in a field failure? 
e Getting caught without replacement parts 
— Pay attention to end-of-life buys 
— Anticipate obsolete: hardware, media, tools, libraries, ... 
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git commit -m “changes” COMMENT DATE 
On oe f CREATED MAIN LOOP & TIMING CONTROL. 

Bis : ENABLED CONFIG FILE PARSING 

MISC BUGFIXES 

CODE ADDITIONS/EDITS 

MORE CODE 

HERE HAVE CODE. 

ARAAAAAA 

ADKFISLKDFISOKLFI 

MY HANDS ARE TYPING WORDS 

HAAAARARAAANDS 











AS A PROJECT DRAGS ON, MY GIT COMMIT 
MESSAGES GET LESS AND LESS INFORMATIVE. 


https://xkcd.com/1296/ 


O RLY? @ ThePracticalDev 
https://goo.gl/pvDMHX CC BY-NC 2.0 
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WHAT WAS THE THE CLOUD WAS A LOT 
INTERNET LIKE IN SMALLER. IT WAS CALLED 
THE OLDEN DAYS, A “MAINFRAME” AND |T 
FOR A DEVELOPER? | | WAS NEAR SACRAMENTO. 
OH, THINGS IT WAS ON THE STATE 
WERE VERY’ | | LANDLINE, 50 THE WHOLE 
DIFFERENT. INDUSTRY PAUSED WHEN 
) THE GOVERNOR HAD TO 
MAKE A PHONE CALL. 


"GIT" WAS ORIGINALLY’ A VAN THAT CIRCLED 
AROUNP GATHERING DATA TAPES To COPY AND 
DISTRIBUTE. WE ALL TOOK TURNS DRIVING IT 
WHEN YOU SALW IT COMING YOU'D BLOW AN 
AIR HORN TO KEQUEST THAT IT PULL OVER. 
THAT'S WHERE “PULL 
REQUEST’ CAME FROM. 





THERE WAS NO MEMORY PROTECTION. IF 
YOU WANTED TO WRITE To AN ADDRESS, 
YOU WOULD CALL AROUND Td ASK 
WHETHER ANYONE ELSE WAS USING IT. 
OFTEN BILL GATES WOULD SAY’ HE WAS, 
EVEN WHEN HE WASNT. THAT'S HOW 
MICROSOFT GOT ITS EARLY FOOTHOLD 


BEFORE TERMINALS, WE ALL USED 
PUNCH CARDS WHICH WERE ORIGINALLY 
DEVELOPED TO CONTROL LOOMS. 
EARLY MAINFRAMES WOULD PRODUCE A 
EVENTUALLY WE GOT THEM TO STOP 
WE HAD ENOUGH SWEATERS. https://m.xked.com/2324/ 
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“Poorly documented legacy code leads to anger. Anger leads to hate. 
Hate leads to spaghetti code.” 
g z 
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Darth Vader is a trademark of Disney, not SciTools. 
https://scitools.com/legacy-code-your-journey-to-the-dark-side/ © 2020 Philip Koopman 13 





